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Task 


“...Review  the  Department’s  T&E  enterprise  to  assess  its  management  and 
utilization  of  both  organic  and  contract  resources  and  infrastructure... to  ensure 
excellence  in  managing  the  enterprise,  reduce  duplication,  and  achieve  savings. 
The  review  should  include... any  other  such  matters  as  the  DBB  deems  relevant.” 

-T&E  TOR,  signed  21  Jan  16 
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Bottom  Line  Up  Front 


■  “Test  &  Evaluation”  (T&E)  is  a  function  of,  and  critical  to, 
the  acquisition  process 

-  True  costs  (dollars  and  people)  are  unknown 

-  Infrastructure  is  not  fully  postured  for  the  future 

-  Different  business  and  risk  decisions  made  from  the  same  test 
data  pressure  cost  and  schedule 

-  Too  many  approvers,  Services’  feel  unable  to  influence  risk 
calculus 

-  Diffuse  risk  accountability  results  in  very  low  tolerance  for  risk 

■  Recommendations 

-  Align  T&E  accountabilities  with  Program  Management 

-  Strengthen  Service  voice  in  risk  decisions 

-  Create  cost  discipline  and  apply  it  as  a  management  tool 

-  Improve  effectiveness  of  T&E  infrastructure 
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Test  and  Evaluation  Ecosystem 


■  Purpose 

-  Part  of  the  overall  DoD  acquisition  process 

-  Assess  fitness  against  requirements  and  performance 

-  Key  acquisition  process  decisions  are  informed  by  test  results 

■  Function 

-  Validate  design  parameters  throughout: 

•  Technology  Maturation  and  Risk  Reduction  Phase  (TMRR)(Between 
Milestone  A&B) 

•  Engineering  and  Manufacturing  Development  Phase  (EMD)(Between 
Milestone  B&C) 

•  Production  and  Development  Phase  (P&D)(Post  Milestone  C) 
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Test  and  Evaluation  Ecosystem 


■  Cost 

-  June  2015  IDA  study  (based  on  many  caveats  and  assumptions) 

•  FY14  -  ~$8.23B 

•  FY16  est.  -  $9.05B 

-  Indirect  costs  unknown 

■  People 

-  Many  Organizations  and  Contractors  -  total  FTEs  unknown 

-  ~26k  Military  /  Civilian  /  Contractor  in  infrastructure 

■  Places 

-  23  MRTFB  ranges  all  over  country 

-  Countless  contractor  facilities  (primarily  for  DT  that  is  conducted 
by  a  contractor  as  a  part  of  an  acquisition) 
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Integrated  Defense  Acquisition,  Technology,  and  Logistics  Life  Cycle  Management  System 
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Infrastructure 


MRTFB  -  23  Sites  Total 


Legend: 

Army,  Navy,  AF,  Defense  Agency 


Cold  Regions 
Test  Center 


NUWC  Keyport  (Nanoose 
&  Dabob  ranges) 

NAWC-WD  China  Lake 


30th  Space  Wing 


NAWC-WD 
Point  Mugu 

412th  Test  Wing 

Tropic  Regions 
Test  Center, 
various  locations 
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Findings:  Process 


■  T&E  is  a  small  but  essential  function  within  the  large  and 
complex  DoD  acquisition  process 

■  The  overall  process  design  integrating  T&E  into  major 
programs  seems  reasonable  and  logical 

-  Methodology  is  well  understood  and  time  tested 

-  All  programs  include  a  T&E  component 

-  The  Test  and  Evaluation  Master  Plan  (TEMP)  is  the  key  T&E 
document  for  a  program 

■  Test  points  and  the  plan  to  validate  them  (via  the  TEMP) 
are  difficult  to  fully  define  early  in  a  program 

-  Fidelity  is  gained  as  the  program  and  technology  matures 

-  Large  programs  may  have  up  to  30  separate  entities  that  must 
approve  the  “plan”  along  the  way 

9  /— ■v 

Final -Approved  by  the  DBB  on  20  October  2016  DEFENSE  BUSINESS  BOARD 


Findings:  Process  (continued) 


■  Overall  T&E  costs  are  unknown  and  not  viewed  as  a  priority 

-  Lack  of  incentive  to  reduce  cost  because  cannot  track  them  directly 

■  Philosophical  differences  between  testing  to  original  requirements  vs.  new 
adversary  capabilities 

■  Accountability  for  risk  trade-off  decisions  is  not  clear  or  consistent 

■  Perception  that  drive  for  zero  risk  is  too  strong  and  uncompromising 

■  Services  can  feel  left  out  of  final  decisions  and  lack  authorities  to  make  final 
“calls”  on  suitability  and  effectiveness 

Lack  of  resources  when  additional  testing  is  required  creates  pressure  on 
budget  and  schedule 

■  DoD  and  industry  are  engaged  in  addressing  the  emerging  challenges  and 
generally  work  well  together;  but  many  view  DoD  process  as  bureaucratic 
and  overly  complex 
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Findings:  Process  (continued) 


■  Strategic  Capabilities  Office  (SCO)  offers  alternatives  to 
the  current  test  approach 

-  New  take  on  the  “development  /  test  /  deployment”  method 

■  Misaligned  incentives  drive  process  participants  to  act  in 
certain  ways 

-  Budget  (PPBE),  Requirements  (JCIDS),  Program  Management, 
T&E 
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Findings:  Infrastructure 


Infrastructure  is  aging  and  unlikely  to  be  feasible  or  cost  effective  testing 
new  or  future  technologies  (e.g.  cyber,  hypersonics) 

-  Legacy  programs  cited  as  reason  for  holding  on  to  old  and  expensive 
assets  -  inadequate  analysis  of  actual  or  opportunity  costs 

-  Causes  overhead  burden  and  need  to  find  other  private  and  public 
users  to  offset  operations  and  maintenance  costs 

Processes  used  for  scheduling  and  testing  are  inconsistent  among  major 
test  facilities 

-  “Works  fine  as  long  as  you  know  the  right  person” 

DoD  building  master  database  of  government-owned  testing  assets 

Management  metrics  for  T&E  infrastructure  are  limited  and  applied 
inconsistently  by  services  and  locations 

Recognize  need  to  move  toward  reducing  physical  testing  through  software 
and  simulation,  but  leadership  disagrees  on  potential  impact  and  progress  is 
slow 
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Findings:  Industry 


■  No  consistent  model  for  DT  and  OT  in  industry 

-  But,  independence  of  T&E  is  valued,  friction 
accepted/encouraged 

■  Product  development  cycles  are  shrinking 

-  Urgency  to  be  “first  to  market”  for  competitive  advantage 

■  Industry  in  general  is  further  along  in  the  application  of 
software  and  simulation  for  T&E 

-  Important  to  reducing  cost  and  time  to  market 

-  Models  are  validated  and  supplemented  by  historical  live  testing 
data  and  select  physical  tests 

■  T&E  costs  are  known  and  examined  routinely 
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Findings:  Industry  (continued) 


■  Successful  companies  have  common  characteristics: 

-  Rely  on  small,  empowered  and  trusted  teams  to  execute  product 
development  and  appropriate  DT  and  OT 

-  Teams  include  leadership  from  business  and  engineering 
organizations  who  together  make  risk  tradeoff  decisions,  with 
business  leader  generally  holding  the  “final  vote” 

-  Apply  extensive  use  of  game  theory  and  probability  analytics  to 
inform  T&E  decisions  including  risk  calculations 

-  Accept  and  incorporate  cost  disciplines  throughout  T&E  process 

-  To  enable  speed  to  market,  they  may  accept  competitive  market 
gaps,  relying  on  next  generation  to  address  them 

•  Requirements  “lock”  and  subsequent  iterative  development,  testing,  and 
fielding 

-  Agreement  going  into  a  program  among  stakeholders  about 
what  successful  achievement  of  requirements  looks  like  and  how 
success  is  measured 
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Recommendations:  Process 


■  Create  appropriate  and  direct  accountabilities  for  T&E 
community,  aligned  with  Program  Management 

-  Should  have  a  stake  in  meeting  budget,  schedule  and 
performance  elements 

-  Will  require  a  greater  involvement  and  commitment  of  T&E 
community  at  the  beginning  and  throughout  program  life-cycle 

-  Definitions  of  “success”  should  be  agreed  between  PM  and  T&E 
community  from  the  beginning  and  with  changes  that  occur  over 
time 

-  Sharply  limit  the  number  and  weight  of  stakeholders  that  have 
veto  power  (stated  or  implied)  over  TEMP  development, 
implementation  and  modification 

•  Clarify  those  with  approval  and  veto  power  vs  those  who  have  advisory  role 
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Recommendations:  Process  (continued) 


■  Strengthen  Service  voice  in  critical  program  decisions 

-  Authorities  should  come  from  the  Service  Secretary  to  the  Service 
Chief  and  designated  subordinates 

-  Supported  by  stronger  more  reasonable  risk  assessment 
determination  as  programs  progress 

•  Today’s  drive  toward  “zero  risk”  is  an  impossible  standard  and  should  not  be 
used  by  the  T&E  community 

•  Adopt  “principled  compromise”  between  absolute  zero-defect  test  doctrine  and 
situational  operating  requirements 

•  “Tolerable”  risk  should  be  stated  and  quantified 

•  Apply  more  consistent  analysis  of  common  data  in  the  T&E  process 

-  Services  are  best  positioned  to  determine  “good  enough”  when 
program  length  results  in  new  threats  that  didn’t  exist  when 
requirements  set 

-  In  effect  the  equivalent  of  the  private  sector  “business”  leaders  in 
terms  of  influence  on  final  fit  for  Service 
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Recommendations:  Process  (continued) 


■  Create  Cost  Discipline  and  Utilize  as  Critical 
Management  Tool 

-  Identify  organization  to  take  responsibility  for  documenting 
current  costs  -  direct  and  indirect,  military,  civilian  and  contractor 

•  Consider  ASD  Research  &  Engineering  (ASD  R&E),  supported  by  OSD 
(Comptroller) 

-  Establish  management  metrics  based  on  these  costs  to  identify 
improvement  areas,  including  where  resources  are  inadequate 
as  well  as  where  resources  are  redundant  or  inefficiently  used 

-  Creates  better  understanding  of  where  money  is  spent  today, 
facilitates  better  decisions  regarding  obsolete  assets  and 
facilities  and  provides  a  path  forward  for  greater  use  of 
simulation  and  software  tools 

-  Do  not  create  new  organization,  but  instead  rely  on  existing 
organizational  elements  to  “own”  their  piece  of  the  cost 
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Recommendations:  Infrastructure 


■  Improve  the  effectiveness  of  T&E  infrastructure  through: 

-  Common  processes  for  enterprise-wide  facility  utilization 

-  Develop  and  apply  common  utilization  metrics 

-  Continue  efforts  to  build  T&E  asset  data  base 

-  Aggressively  invest  in  and  utilize  software  based  simulation  to 
reduce  percentage  of  T&E  relying  on  physical  testing 

-  Adopt  existing  privatization  tools  to  help  recapitalize  T&E  and 
reduce  the  financial  burden  and  risk  of  future  needs  of  the 
government 

-  Seek  ways  to  integrate  the  methodological  approach  to  agile 
development  and  testing  being  led  today  by  the  Strategic 
Capabilities  Office 
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